Skip to content

Bump maven and PySpark to latest#80

Merged
trivialfis merged 10 commits into
dmlc:mainfrom
trivialfis:bump-pyspark
Mar 23, 2026
Merged

Bump maven and PySpark to latest#80
trivialfis merged 10 commits into
dmlc:mainfrom
trivialfis:bump-pyspark

Conversation

@trivialfis

Copy link
Copy Markdown
Member

No description provided.

@trivialfis trivialfis temporarily deployed to containers-staging March 9, 2026 19:59 — with GitHub Actions Inactive
@trivialfis trivialfis changed the title [WIP] Bump PySpark to 4.0. [WIP] Bump PySpark to latest (>=4.0). Mar 9, 2026
@trivialfis trivialfis changed the title [WIP] Bump PySpark to latest (>=4.0). [WIP] Bump maven and PySpark to latest Mar 9, 2026
@trivialfis trivialfis temporarily deployed to containers-staging March 9, 2026 20:13 — with GitHub Actions Inactive

@medb medb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to bump Java version to 17 as this is min supported version by Spark 4.

@trivialfis trivialfis temporarily deployed to containers-staging March 11, 2026 16:17 — with GitHub Actions Inactive
@trivialfis trivialfis temporarily deployed to containers-staging March 11, 2026 22:07 — with GitHub Actions Inactive
@trivialfis trivialfis temporarily deployed to containers-staging March 15, 2026 15:24 — with GitHub Actions Inactive
@trivialfis trivialfis temporarily deployed to containers-staging March 15, 2026 16:18 — with GitHub Actions Inactive
@trivialfis trivialfis temporarily deployed to containers-staging March 15, 2026 16:26 — with GitHub Actions Inactive
Comment thread containers/conda_env/aarch64_test.yml Outdated
Comment on lines +31 to +32
- grpcio # used by spark connect
- grpcio-status

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of installing grpcio manually we can use connect extension in pyspark package to pull all necessary dependencies automatically: pyspark[connect].

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The environment file is for conda, which doesn't support the bracket notation: https://anaconda.org/channels/conda-forge/packages/pyspark/overview

Installing it through pypi can work, but I'm not sure if I want to bring dependencies from pypi and mix them with packages from conda.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, it does not worth it to use pip in conda env

@trivialfis trivialfis temporarily deployed to containers-staging March 17, 2026 11:55 — with GitHub Actions Inactive
@trivialfis trivialfis changed the title [WIP] Bump maven and PySpark to latest Bump maven and PySpark to latest Mar 23, 2026
@trivialfis trivialfis merged commit 0b797b3 into dmlc:main Mar 23, 2026
24 checks passed
@trivialfis trivialfis deleted the bump-pyspark branch March 23, 2026 21:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants